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study was able to find documentation of 178 of them on the states' Web sites 
or direct links. Furthermore, an additional 216 indicators not identified by 
the ECS were located. The study found that it was difficult to determine the 
extent to which any of these indicators were used in making determinations 
for consequences. Most often, assessment scores served as the only indicator 
used for making this determination. The study found that although students 
with disabilities may be included in assessment, their scores may not be 
included in the accountability system, and most states did not specify 
whether students with disabilities were included when defining the indicators 
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Executive Summary 



This report is the first by the National Center on Educational Outcomes (NCEO) to examine the 
extent to which students with disabilities are included in accountability systems of states in 
which there are high stakes for educational systems. Conducted through a Web search of states’ 
education Web sites and direct links to their Web sites, we examined the indicators on which 
states collect information (e.g., student indicators, staff indicators, program indicators, and use 
of resources indicators) as well as the consequences provided by the states (e.g., rewards, assis- 
tance, or sanctions). 

Of the 280 indicators listed for these states according to the Education Commission of the 
States (ECS), NCEO was able to find documentation of 178 of them on the states’ Web sites or 
direct links. Furthermore, NCEO found an additional 216 indicators not identified by ECS. 
However, it was difficult to determine the extent to which any of these indicators were used in 
making determinations for consequences. Information for many indicators was collected and 
reported but not included in the actual formulas for determining rewards, assistance, or sanc- 
tions. Most often, assessment scores served as the only indicator used for making this determi- 
nation. 

Five states had accountability formulas that could be identified in information found on state 
Web sites. Louisiana, Mississippi, New York, North Carolina, and Ohio listed their indicators 
and criteria as well as how they are used to determine the consequences for states. From the 
information found, other states may have some of these components of an accountability sys- 
tem, but did not provide the level of detail on their Web sites needed to determine how their 
systems were implemented. 

This study highlighted that not only was it difficult to determine exactly what indicators fac- 
tored into overall accountability systems, but it was also difficult to find evidence that students 
with disabilities were included in accountability systems. Although students with disabilities 
may be included in assessments, their scores may not be included in the accountability systems, 
and most states did not specifically state whether students with disabilities were included when 
defining the indicators used. 
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Overview 



Educational accountability is gaining attention and influence as state and local policymakers 
implement sweeping accountability reforms. States are increasingly relying on large-scale 
assessments to measure student performance and to determine sanctions and rewards (Education 
Commission of the States, 1999; Olson, Bond, & Andrews, 1999). The Web sites of major 
organizations, such as the Council of Chief State School Officers (CCSSO: http://www.ccsso.org) 
and the Education Commission of the States (http://www.ecs.org) reflect this emphasis in content 
on their Web pages and their on-line reports. 

Approximately half the states now use, or will soon use, tests in making decisions about student 
promotion (see Quenemoen, Lehr, Thurlow, Thompson, & Bolt, 2000) and high school graduation 
(see Guy, Shin, Lee, & Thurlow, 1999). All but two states have statewide assessments in place, 
many with more than one assessment (Olson, Bond, & Andrews, 1999). In the 2000 legislative 
sessions, over 300 bills were introduced in 41 states that dealt with statewide assessments 
(Coleman, Heller, & McNeil, 2000; see also the Web site for the National Conference of State 
Legislatures, http://www.ncsl.org/programs/educ/kl21ink.htm ). 

Not only are states increasingly attaching high stakes consequences to students for their 
performance on assessments (e.g., no diploma, no promotion to the next grade), but states are 
also attaching significant consequences to various aspects of the educational system (e.g., to 
teachers, schools, and administrators). This approach, known as system accountability, occurs 
when educators, administrators, schools, or districts are held responsible and consequences are 
assigned to them for various student performance indicators. For example, school accreditation 
may be based on test scores; teachers may receive cash rewards for student performance; or, 
administrators may be either sanctioned or rewarded based on school improvement, which in 
turn is measured to a large extent by student performance. 

Within system accountability, there are different levels of consequences - referred to as the 
“stakes” of the assessment. At one end is simply publishing test scores - usually considered to 
be low stakes. While it might be considered “positive feedback” to have a school that performed 
well on test scores, or conversely, might prove an embarrassment for schools or districts that 
performed poorly, these consequences are relatively minor. In contrast, if teachers are assigned 
to “less prestigious” schools or receive cash bonuses based on student performance, the stakes 
are considerably higher. Similarly, if principals receive increased school funds or lose their jobs 
based on how students in the school perform, the stakes are very high. 

In addition to obvious “rewards” and “punishments,” there is another type of consequence 
currently used in accountability systems. This type of consequence involves assistance to the 
system and is generally considered to be a much less significant consequence. An example of 
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assistance might include the provision of additional resources, such as monetary grants or 
specially trained personnel to schools that receive a “warning.” The distinction between 
“assistance” and “sanctions” is often subtle, at best. The “assistance” phase sometimes serves 
as a probationary period prior to more intense intervention or reconstitution of a school or 
district. 

The Education Commission of the States (1999) has documented the specific indicators that 
states are using in their accountability systems. Although an array of indicators is listed for 
most states, it is unclear how they are used. Many questions remain about current and proposed 
accountability systems. Which indicators are truly high stakes? Which are simply included in a 
“report card” for a school or district? Are the indicators weighted in some manner or incorporated 
in a formula to develop an accountability measure? Are there positive rewards for successful 
schools and systems, assistance for struggling schools and systems, or sanctions for those who 
are not performing as expected? 

In addition to these basic questions about state accountability systems, there are also questions 
that focus on specific groups of students. With the reauthorization of the Individuals with 
Disabilities Education Act in 1997 (IDEA 97), states and districts must include students with 
disabilities in their assessments. They are also required to report on the performance of students 
with disabilities, both aggregated with other students, and disaggregated. 

Several analyses from before and after the enactment of IDEA 97 indicated that states have 
started to include students with disabilities in their assessments only recently (McGrew, Thurlow, 
Shriner, & Spiegel, 1992; Thompson & Thurlow, 1999), and that many states are not yet reporting 
the results (Thurlow, Langenfeld, Nelson, Shin, & Coleman, 1998; Thurlow, Nelson, 
Teelucksingh, & Ysseldyke, 2000) even though they are required to do so by federal law. It is a 
logical follow-up question to ask about the extent to which the increasing participation of students 
with disabilities in assessments translates into their inclusion in accountability systems. 

Of course, there are many reasons for including students with disabilities in accountability 
systems (Thurlow, Elliott, & Ysseldyke, 1998). If these students are not included in accountability 
systems, it means that roughly 10% of the student population is excluded from accountability. 
Students who are left out of assessments and accountability systems tend to not be considered 
when reform efforts are being designed and implemented. Furthermore, the education system 
does not obtain an accurate picture of education if a significant portion of students is excluded, 
nor is it able to make “fair” comparisons. Participation of students in accountability systems 
seems to promote higher expectations for those students who are included, and consequently, 
may be assumed to exclude from high expectations those students who are not included. 

The purpose of this research was to examine the extent to which students with disabilities are 
included in accountability indicators in those states in which there are high stakes for systems. 
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To do this, we had to first decipher the accountability systems that states use - this involved 
identifying both the indicators that are part of the accountability system and the “formula” that 
is used in each state to combine the indicators to determine an accountability score. We restricted 
our search to accountability indicators and formula used to determine system-level consequences 
and did not search for formula developed exclusively to meet Title I requirements for adequate 
yearly progress. From the indicators and formula identified, we examined the extent to which 
students with disabilities were included in the state’s accountability system. 



Initially, we used the document Education Accountability Systems in 50 States (Education 
Commission of the States/ECS, 1999) to determine which states we would include in our analysis. 
We selected states that ECS had listed as having a “statute” or “regulation” in the Rewards or 
Sanctions column of its Table I. There were 38 states that met this criterion. The ECS document 
also included information about which indicators were used by the various states (see ECS 
Table 5). We divided in half the list of states that had high stakes accountability systems for 
schools or districts, then two of us examined Web sites to verify and expand on the information 
that was included in the ECS report. 

We used the Achieve Web site (http://www.achieve.org) “State Links” page as a quick link to 
the Web site for the Department of Education of each state. From this site, we also located other 
state offices (e.g., accountability, assessment, legislature). 

We also used the Council of Chief State School Officers (CCSSO) Web site (http:// 
www.ccsso.org) section on “Council Projects” where we located the Accountability Systems 
Profiles. (The direct Web address is http://www.ccsso.org/introprofile.html ). According to this 
site, the Resource Center on Educational Equity of the Council of Chief State School Officers 
(CCSSO), in collaboration with the New England Comprehensive Assistance Center, is 
developing profiles of several state’s education accountability systems. 

When we checked the site, there was information about the following 13 states: Alabama, 
Connecticut, Georgia, Illinois, Kentucky, Maryland, Massachusetts, New Jersey, North Carolina, 
Oregon, Rhode Island, Texas, and Vermont. The profiles included information about each state’s 
content and performance standards, statewide student assessment, and criteria used to determine 
whether schools or districts are eligible for rewards or are in need of improvement. The site also 
contained information about how school and district performance is publicized, as well as details 
about the provision of rewards, assistance, and sanctions. There was often helpful information 
about the inclusion of students with disabilities in the assessment systems, as well as information 
about provisions for English Language Learners. 
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The Resource Center on Educational Equity page of the CCSSO Web site indicated that the 
information for the 13 profiled states was gathered from various sources, including CCSSO 
reports, state statutes and regulations, and information posted on state education agency Web 
sites. The Center’s initial drafts were reviewed and revised by state education agency staff. We 
used the information from CCSSO’s state profiles only after independently verifying it on state 
Web sites. 

At each state’s Web site we reviewed state accountability reports, report cards, policy guidelines, 
and other relevant items to identify indicators, consequences, and accountability formula. This 
review was conducted in 1999-2000, with all reviews completed by July 2000. After we 
completed our initial examinations of Web site information on states, the two lists of states 
were traded, so that the information could be verified. In some instances, there was little change 
from what the first examination had indicated. In other cases there was completely new and 
sometimes contradictory information obtained during the second examination of the Web site. 
This report contains the more recent version of those sites. 



It became very evident during our verification process that the information we sought was a 
moving target. It changed rapidly due to ongoing legislative processes, subsequent rule-making 
procedures that follow legislative sessions (in order to implement the legislation just passed), 
and states’ action taken in response to federal legislation. Our focus was on Web-based 
information, and we found a significant disparity between the level of sophistication and quality 
of information found on various states’ Web sites. Some seemed to have frequent additions or 
modifications, and others seemed to be fairly rudimentary. Links to other Web sites were 
prominent and user-friendly on some sites, and unavailable or very obscure on others; sites 
were frequently “down” or “temporarily under construction.” For example, important documents 
found during the first review were often moved or unavailable during the second review. 
Sometimes Web sites would change dramatically within a few days. 

We collected information on the indicators used for accountability purposes, the formulas for 
calculation of school or district scores for accountability purposes, the consequences of the 
accountability systems (e.g., rewards, assistance, or sanctions), and the inclusion of students 
with disabilities in the accountability systems’ indicators and calculations. 
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Accountability Indicators 



Table 1 summarizes the numbers of indicators that were identified in the ECS document and in 
our Web search. Of the 280 specific indicators listed in the ECS document, we were able to 
verify 178 of them when examining the State Departments of Education Web sites and direct 
link sites. In other words, we found, on average, only 64% of the indicators that ECS found 
using written reports. (Values ranged from 15% for Massachusetts to 100% for Alaska, Colorado, 
Kansas, Kentucky, New Jersey, Oklahoma, Rhode Island, South Carolina, Tennessee, Vermont, 
and West Virginia.) 

We also discovered on State Department Web sites 216 additional indicators used by states for 
their accountability systems. While the majority of indicators listed were for reporting purposes 
only (i.e., they are documented in school and district report cards), some of these indicators are 
used in accountability formulas to calculate school or district scores to determine access to 
rewards or assistance or the receipt of sanctions. 

Although a state may have indicated on its Web site that it uses a large number of indicators, we 
found that it was often the case that the majority of the indicators were of the same type (e.g., all 
assessment scores). Thus, looking at only the number of indicators does not reflect the nature of 
the accountability system. To get a different picture, we looked at the spread of types of indicators, 
and compared these to the types found in the ECS analysis. The results of this comparison are 
provided in Appendix A. As is evident, relatively few different types of indicators are being 
used by states, and the consistency between the types our search revealed and those ECS identified 
are relatively similar. 



Accountability Formula 

Five states (Louisiana, Mississippi, North Carolina, New York, and Ohio) had accountability 
formulas using indicators to calculate school or district scores listed on their Web sites. The 
formulas are quite diverse, from those that simply divide the number of students reaching a 
given proficiency level by the total number of students taking the test, to those that have included 
factors for regression to the mean. 

Other states have information on their Web sites that were close to being formulas, but are too 
vague to actually determine how accountability is calculated. For example, Indiana noted that a 
school receives a monetary incentive if the school has demonstrated improvement over the 
average of its scores for the past three years in two or more areas (with the areas defined as 
attendance rates, math scores, language arts scores, and ISTEP + total battery scores). Similarly, 
Pennsylvania indicates on its Web site that it gives rewards based on improvements in 
achievement rates over the past two year average, or higher graduation rates or attendance rates 
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Table 1. Indicators in State Accountability Systems 



State 


ECS Count 
of Indicators 


NCEO 

Confirmed 

ECS 

Indicators 


Additional 
Indicators on 
Web site 


Accountability 
Formula on 
Web site* 


Alabama 


5 


3 


5 




Alaska 


7 


7 


3 




Arizona 


6 


4 


1 2 




Arkansas 


7 


3 


3 




Colorado 


1 


1 


0 




Connecticut 


1 3 


5 


1 2 




Delaware 


1 1 


2 


0 




Florida 


7 


5 


1 




Georgia 


8 


3 


0 




Illinois 


1 2 


9 


0 




Indiana 


1 1 


2 


0 




Iowa 


0 


— 


3 




Kansas 


1 2 


1 2 


0 




Kentucky 


5 


5 


0 




Louisiana 


1 3 


3 


0 


■ 


Maryland 


5 


3 


6 




Massachusetts 


1 3 


2 


2 




Michigan 


4 


1 


0 




Mississippi 


0 


— 


37 


■ 


Missouri 


1 2 


1 0 


8 




Nebraska 


0 


— 


1 6 




Nevada 


1 3 


1 1 


10 




New Jersey 


1 1 


1 1 


10 




New Mexico 


9 


7 


4 




New York 


7 


3 


3 


■ 


North Carolina 


6 


1 


0 


■ 


Ohio 


1 2 


9 


0 


■ 


Oklahoma 


8 


8 


6 




Oregon 


7 


3 


6 




Pennsylvania 


3 


2 


2 




Rhode Island 


6 


6 


7 




South Carolina 


2 


2 


10 




Tennessee 


3 


3 


1 




Texas 


7 


7 


1 1 




Vermont 


1 1 


1 1 


4 




Virginia 


5 


4 


7 




Washington 


1 1 


3 


1 




West Virginia 


7 


7 


26 




Totals 


280 


178 


216 


5 



* Specific formulae are in Appendix B. 
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over a criteria set each year (but the criteria were not provided). The state also indicated that 
dropout rates are collected but not counted in the reward system. Finally, Kentucky’s Web site 
clearly indicates that it has an accountability index that is a statistic that combines a school’s 
academic, nonacademic, and other “appropriate data,” but how these elements are combined is 
not evident. Dropout criteria also must be met, although the specific criteria are not stated. 
Kentucky also indicates that it uses a regression model to compare expected growth with actual 
growth in determining when a reward is given. 

Of the five states' formulas, only one specifically mentions students with disabilities (and another 
mentions English Language Learners). In the four states that do not mention students with 
disabilities, the implication is that all students are included in the same way. The one formula 
that did identify students with disabilities actually factored in a weight that decreases the impact 
of the scores of students with disabilities on the resulting accountability index. The rationale for 
this approach is that special education students might take longer to achieve standards because 
of lost opportunities to learn. Detailed descriptions of the formula are provided in Appendix B. 



Consequences 

Table 2 summarizes the nature of consequences of school or district performance in each of the 
states examined. It is important to note that this table is a point-in-time snapshot. Several states 
indicated that they are currently changing their existing accountability system or are implementing 
new components to their accountability systems. Currently under development are reward 
systems for five states, assistance programs for two states, and sanctions for three states. 

Insufficient information was available on some of the Web sites to determine whether any 
rewards, assistance, or sanctions were given. Eleven states did not include any information 
about rewards for high achieving schools or districts. Seven states did not include any information 
about assistance programs for low achieving schools or districts on their Web sites. Eight states 
did not include any information about sanctions for chronically low achieving schools or districts. 

Of the 38 states listed as having high stakes system accountability, information on consequences 
could be found on the Web sites of 34 of them. Of the 34 states with some consequence in place 
for performance, 17 had all three components: rewards, assistance, and sanctions. An additional 
seven states had both assistance and sanctions. Of the 34 states that provided information on 
their accountability components, seven had no form of sanctions, seven had no form of assistance, 
and nine had no form of reward. 
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Table 2. Nature of Consequences in State Accountability Systems 



State 


Rewards for 
High Achieving 
Schools 


Assistance for 
Low Achieving 
Schools 


Sanctions for 
Low Achieving 
Schools 


Alabama 


None Specified 


Yes 


Yes 


Alaska 


None Specified 


Yes 


None Specified 


Arizona 


None Specified 


None Specified 


Yes 


Arkansas 


Under Development 


No 


Yes 


Colorado 


Yes 


Yes 


Yes 


Connecticut 


Yes 


Yes 


Yes 


Delaware 


Under Development 


Under Development 


Under Development 


Florida 


Yes 


Yes 


Yes 


Georgia 


Yes 


Yes 


Yes 


Illinois 


Yes 


Yes 


Yes 


Indiana 


Yes 


None Specified 


None Specified 


Iowa 


None Specified 


None Specified 


None Specified 


Kansas 


Yes 


Yes 


Yes 


Kentucky 


Yes 


Yes 


Yes 


Louisiana 


Yes 


Yes 


Yes 


Maryland 


Yes 


Yes 


Yes 


Massachusetts 


Under Development 


Under Development 


Under Development 


Michigan 


None Specified 


None Specified 


None Specified 


Mississippi 


Yes 


Yes 


Yes 


Missouri 


Yes 


None Specified 


None Specified 


Nebraska 


Yes 


No 


No 


Nevada 


None Specified 


Yes 


Yes 


New Jersey 


Yes 


Yes 


Yes 


New Mexico 


Yes 


Yes 


Yes 


New York 


Under Development 


Yes 


Yes 


North Carolina 


Yes 


Yes 


Yes 


Ohio 


Yes 


Yes 


Yes 


Oklahoma 


None Specified 


Yes 


Yes 


Oregon 


Yes 


Yes 


Yes 


Pennsylvania 


Yes 


None Specified 


None Specified 


Rhode Island 


None Specified 


Yes 


None Specified 


South Carolina 


Yes 


Yes 


Yes 


Tennessee 


None Specified 


Yes 


Yes 


Texas 


Yes 


Yes 


Yes 


Vermont 


None Specified 


Yes 


Yes 


Virginia 


Yes 


None Specified 


None Specified 


Washington 


Under Development 


Yes 


Under Development 


West Virginia 


None Specified 


Yes 


Yes 


Totals 


Yes 22 

Under Development 5 

None Specified 11 

No 0 


Yes 27 

Under Development 2 

None Specified 7 

No 2 


Yes 26 

Under Development 3 

None Specified 8 

No 1 
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Rewards 



Of the states that provided information on their accountability components, 22 states indicated 
that they provided rewards for schools or districts with high achievement (based on their 
indicators). Table 3 summarizes the types of rewards offered by these states. Details of the 
rewards provided by each state are available in Appendix C. 

Most states provide some form of tangible or monetary rewards. For example, Nebraska provides 
Quality Education Incentive Payments and Georgia provides Pay for Performance awards. 
These monetary rewards are most commonly used to purchase equipment, pay for professional 
development, or make possible other improvements in the schools’ educational processes. 

A majority of states also have some process for formal recognition or praise of schools or for 
districts with superior performance. Examples include banners, certificates, and plaques. In all 
but two of the states providing recognition or praise, there is also the use of tangible or monetary 
rewards. Only Mississippi and Virginia use recognition or praise without tangible or monetary 
rewards. 

Six states provide some form of additional decision-making control or exemption from policy 
guidelines for districts or schools that demonstrate high achievement. For example, Florida 
states that schools rated high are eligible for deregulated status and increased budgetary control, 
while Kentucky allows high-performing schools to be exempted from certain reports, paperwork 
requirements, and administrative regulations. 

Two states are coded as “Other” in Table 3 because of their unique situations. In Colorado, 
decisions about rewards are made at the district level. Kansas uses its indicators to determine 
school accreditation; schools with high student achievement and continual academic 
improvement receive a special accreditation. For five states (Arkansas, Delaware, Massachusetts, 
New York, and Washington), the process for determining rewards for high achieving schools is 
under development. 



Assistance 

Twenty-six states indicated that they had assistance for schools or districts that had low 
achievement according to their indicators. Table 4 summarizes the types of assistance offered 
by the states. Details on the assistance provided by each state are presented in Appendix D. 

Most states that provide assistance either require schools or districts to develop improvement 
plans or the states provide assistance in developing these improvement plans. For example, 
Oregon districts and schools identified for improvement are required to develop Improvement 
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Table 3. Rewards for High Achieving Schools 



State 


Tangible or 
Monetary 
Rewards 


Recognition 
or Praise 


Policy 
Control or 
Exemption 


Under 

Development 


Other 


Arkansas 








■ 




Colorado 










■ 3 


Connecticut 


■ 










Delaware 








■ 




Florida 


■ 


■ 


■ 






Georgia 


■ 


■ 








Illinois 






■ 






Indiana 


■ 


■ 








Kansas 










■ b 


Kentucky 


■ 




■ 






Louisiana 


■ 


■ 








Maryland 


■ 


■ 








Massachusetts 








■ 




Mississippi 




■ 


■ 






Missouri 


■ 


■ 








Nebraska 


■ 










New Jersey 


■ 


■ 








New Mexico 


■ 


■ 








New York 








■ 




North Carolina 


■ 


■ 








Ohio 






■ 






Oregon 


■ 


■ 








Pennsylvania 


■ 










South Carolina 


■ 


■ 








Texas 


■ 


■ 


■ 






Virginia 




■ 








Washington 








■ 




Totals 


1 6 


1 4 


6 


5 


2 



a Decisions are made at the school or district level or on an individual basis. 
b High Performing schools are designated through a Quality Performance Accreditation. 



Plans with descriptions of their assessment of problem areas, proposed activities to be undertaken, 
and a timeline for achieving these goals. New Mexico selects the lowest performing schools to 
receive assistance from the State Board of Education in creating a School Improvement Plan. 
Often these plans include outcome criteria against which a school or district’s performance is 
measured, or they may contain binding agreements for changes to be implemented. 

The same number of states (but not necessarily the same states) that require improvement plans 
provide some form of non-monetary assistance. Seventeen states have both improvement plans 
and non-monetary assistance. Examples of non-monetary assistance include state assistance 
teams that conduct site visits and provide recommendations, technical assistance with data 
analysis, staff training and consultation/mentoring by specifically designated teachers, 
administrators, or teams of educators. 
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Table 4. Assistance for Low Achieving Schools 



State 


Improvement 

Plans 


Non-Monetary 

Assistance 


Monetary 

Assistance 


Under 

Development 


Other 


Alabama 


■ 


■ 








Alaska 


■ 










Colorado 










■ a 


Connecticut 






■ 






Delaware 








■ 




Florida 




■ 


■ 






Georgia 


■ 


■ 


■ 






Illinois 


■ 


■ 








Kansas 




■ 








Kentucky 


■ 


■ 


■ 






Louisiana 


■ 


■ 








Maryland 




■ 








Massachusetts 








■ 




Mississippi 


■ 


■ 








Nevada 


■ 










New Jersey 


■ 


■ 








New Mexico 


■ 


■ 








New York 


■ 










North Carolina 


■ 


■ 


■ 






Ohio 


■ 




■ 






Oklahoma 




■ 


■ 






Oregon 


■ 


■ 








Rhode Island 


■ 


■ 


■ 






South Carolina 


■ 


■ 


■ 






Tennessee 


■ 


■ 








Texas 


■ 


■ 








Vermont 


■ 


■ 








Washington 


■ 


■ 








West Virginia 


■ 


■ 








Totals 


21 


21 


9 


2 


1 



a Decisions are made at the school or district level or on an individual basis. 



Nine states provide some form of monetary assistance, which may be coupled with improvement 
plans, non-monetary assistance, or both. Ohio provides Continuous Improvement Development/ 
Implementation Grants for districts to design and implement comprehensive plans in conjunction 
with their continuous improvement plans. South Carolina offers targeted grants for professional 
development, alternative schools, homework centers, and modified school days (i.e., extended 
year or day schools). 

Colorado is coded as “Other” in Table 4 because decisions about whether a school or site will 
receive some form of monetary or non-monetary assistance are made locally. In Delaware and 
Massachusetts, the process of determining rewards, assistance, or sanctions is under development. 
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Sanctions 



Twenty-six states indicated that they had sanctions for schools or districts with chronic low 
achievement. Table 5 summarizes the types of sanctions administered by these states. Appendix 
E gives details of the various sanctions. 

Twenty-one states reported using some process to notify the public of low achieving schools. 
While many more states may identify schools as low achieving or provide information on schools’ 
poor performance, public notification as coded in Table 5 consisted of a more formal process 
whereby a label or level was assigned to the school that clearly identified it as low achieving. 
For example, Alabama labels schools and systems as Academic Alert Levels 1, 2, or 3, Arkansas 
labels schools as High Priority, Alert Status, Low Performing Status and Academic Distress, 
and Texas labels campuses as Low-Performing and districts as Academically Unacceptable. In 
addition, states that have a formal notification process were coded in this category of sanctions. 
For example, in Kansas each school issues a report of Accreditation Deficiencies that includes 
its plans to correct the deficiencies; these are reported to the communities in their primary 
languages. 

Fifteen states reported enforcing state mandates or assuming some managerial control of low 
achieving districts or schools. Examples include mandatory assistance, administrative orders 
from Commissioners, and state takeovers of schools. In all but two of these states (New Jersey 
and Oklahoma), mandates or management sanctions accompanied the public notification. 

Twelve states reported the possibility of reorganizing staff or restructuring schools to remedy 
chronically low achieving schools. For example, Tennessee may remove the superintendent 
and board members from office, New Jersey may dismiss or demote school staff, and Connecticut 
may close and reconstitute schools. 

Ten states require schools to pay for the transfer of their students to better performing schools 
or the state pulls the funding for students. For example, Nevada requires schools to pay for 
students to be transferred to the nearest fully accredited school. 

Six states will pull the official status of low achieving schools. Mississippi may withdraw the 
accreditation status of chronically low performing schools, while Illinois may “non-recognize” 
a school or district that may then be dissolved and realigned with other districts. 

Four states are coded as “Other” in Table 5. Both Colorado and Oregon make decisions either 
locally or on an individual basis. Georgia uses a variety of sanctions involving either schools 
raising funds to support corrective actions or the state filing a lawsuit. New Mexico identifies 
schools with accreditation deficiencies, which must be publicly reported. 
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Table 5. Sanctions for Low Achieving Schools 



State 


Public 

Notification 


State 

Mandates or 
Management 


Restructure 
or Reorganize 
Staff 


Lose 
Students 
or Funds 


Lose 

Official 

Status 


Under 

Develop- 

ment 


Other 


Alabama 


■ 


■ 












Arizona 


■ 














Arkansas 


■ 


■ 












Colorado 
















Connecticut 


■ 


■ 


■ 


■ 








Delaware 












■ 




Florida 


■ 


■ 


■ 










Georgia 


■ 


■ 


■ 








■ b 


Illinois 


■ 


■ 


■ 


■ 


■ 






Kansas 


■ 














Kentucky 








■ 








Louisiana 


■ 




■ 


■ 


. ■ 






Maryland 






■ 










Massachusetts 












■ 




Mississippi 


■ 








■ 






Nevada 


■ 


■ 




■ 








New Jersey 


■ 


■ 


■ 


■ 








New Mexico 














■ c 


New York 










■ 






North Carolina 


■ 


■ 


■ 




■ 






Ohio 


■ 


■ 












Oklahoma 


■ 


■ 


■ 


■ 








Oregon 


■ 












■ Q 


South Carolina 


■ 




■ 










Tennessee 


■ 


■ 


■ 










Texas 


■ 


■ 




■ 








Vermont 


■ 


■ 


■ 


■ 








Washington 












■ 




West Virginia 


■ 


■ 




■ 


■ 






Totals 


21 


1 5 


1 2 


1 0 


6 


3 


4 



0 Decisions are made at the school or district level or on an individual basis. 

b The state may increase the school system’s local fair share to finance necessary corrective actions, require the 
local school system to raise funds from local revenue sources to finance the corrective actions, or file a civil 
action in county court to determine if any local school board member or local superintendent has prevented or 
delayed the corrective action plan implementation. 
c Although sanctions were not specified, indicators are used in determining a school’s accreditation. 



Students with Disabilities 

The inclusion of the assessment scores of students with disabilities in accountability systems of 
various states is represented in Table 6. More detailed description of the inclusion/exclusion 
policies of individual states can be found in Appendix F. 

Seven states seem to include all students with disabilities in their accountability systems. One 
state says all are included, but there was no confirming information on the Web site to validate 
or contradict that information. Of the remaining six states, five have documented evidence that 
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Table 6. Inclusion of Assessment Scores for Students with Disabilities in Accountability Systems 



State 


Unclear 

Inclusion/ 

Exclusion 


All SWD Included 


All SWD 
Excluded 


Defined Subset of SWD Excluded 


Not 

Docum- 

ented 


Documented 


Unclear 

Exclusion 


Non- 

Standard 

Accommo- 

dation 


Other 


Alabama 








■ 








Alaska 




■ 












Arizona 












■ 




Arkansas 










■ 






Colorado 


■ 














Connecticut 










■ 






Delaware 












■ 




Florida 










■ 






Georgia 










■ 






Illinois 






■ 










Indiana 












■ 




Iowa 


■ 














Kansas 










■ 






Kentucky 






■ 










Louisiana 






■ 










Maryland 


■ 














Massachusetts 


■ 














Michigan 














■ b 


Mississippi 












■ 


■ C 


Missouri 






■ 










Nebraska 


■ 














Nevada 












■ 




New Jersey 












■ 




New Mexico 






■ 










New York 






■ 










North Carolina 


■ d 














Ohio 


■ 














Oklahoma 


■ 














Oregon 










■ 


■ 




Pennsylvania 










■ 




■ ' 


Rhode Island 










■ 






South Carolina 










■ 






Tennessee 








■ 








Texas 


■ 














Vermont 


■ 














Virginia 


■ 














Washington 










■ 






West Virginia 


■ 














Totals 


1 2 


1 


6 


2 


10 


7 


3 



a This accountability system is currently in place for grades K-8. Grades 9-12 will be added in 2001 . 
b One accreditation criterion requires a 95% participation rate allowing exclusions of 5% of students. 
c Scores may be excluded from students who are expected to eventually meet the same basic curriculum objectives, but 
due to their educational delays, are not expected to meet them for that school year. 
d Students with disabilities are counted in the participation rates for a subtest even if they are excluded from taking it but 
do participate in the other subtests. 

e Reasons allowed for exempting students with disabilities include parent request, IEP status, LEP status, extended 
absence, withdrawal, of other reasons. 
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indeed all students with disabilities count. They are Illinois, Kentucky, Missouri, New Mexico, 
and New York. For example, Illinois clearly indicates that if a student (EEP or non-EEP) takes a 
state assessment and passes the “attemptedness” criterion, that student’s score will be included 
in the school, sub-district, district, and state averages and data computations. Beginning in 
1999, the option to “bubble out” a student’s score (allowing the student to take the state assessment 
and excluding that score from the school, district, and state aggregations) was no longer allowed. 
The seventh state, Louisiana, has a system of inclusion of all students with disabilities currently 
in place for grades K-8 and indicates that grades 9-12 are scheduled to be added in 2001. 

Two states (Alabama and Tennessee) indicate that students with disabilities are excluded from 
their accountability systems. In Alabama, the Web site emphasized the inclusion of students 
with disabilities in assessments, but went on to indicate that special education students’ scores 
were not included in classifying a school or system as Academic Clear, Academic Caution, or 
Academic Alert. On the Web site for Tennessee, there was no definitive information, but there 
was a related link which revealed the fact that any student who is eligible for special education 
services under federal law would not be included as a part of its value-added assessment system. 

More commonly, a defined subset of students with disabilities is excluded. For example, based 
on information on the Web it appears that seven states use non-standard accommodations as the 
determining factor of whether a student with disabilities is included in the accountability system. 
Arizona, Delaware, Indiana, Nevada and New Jersey use non-standard accommodations as the 
sole criteria. In addition to using non-standard accommodations as a reason for exemption, 
Mississippi also permits scores to be excluded for students who are expected to eventually meet 
the same basic curriculum objectives, but due to their educational delays are not expected to 
meet them for that school year. Oregon mentioned non-standard accommodations as a reason 
for exclusion, but was unclear as to whether there were additional criteria for exclusion as well. 

Three states (Michigan, Mississippi, and Pennsylvania) use various other factors to determine 
inclusion or exclusion. An example of this is Pennsylvania, which may exclude students with 
disabilities for a number of reasons including parent request, EEP status, LEP status, extended 
absence, and withdrawal from school. 

Ten states exclude a subset of students with disabilities, but it is unclear in information provided 
on the Web site of how that subset is determined. En Florida, for example, schools are required 
to assess at least ninety percent of all eligible students, but it is not clear how they determine 
which students are excluded. En Georgia, students are excluded if their assessments are coded in 
a particular manner. En South Carolina, students who are included in statewide testing are included 
in the accountability system. However, the exemption rates for students with disabilities in 
1998 exceeded six percent of the student population and represented approximately 60% of the 
special education students. 
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In 12 states it simply was not possible to determine whether all students with disabilities are 
included. For example, Colorado makes no direct reference to the inclusion of special needs 
students. Massachusetts indicates that it will include all public school students in the 
Massachusetts Comprehensive Assessments (MCA) and the grade 3 reading test, but its 
accountability system is still listed as under development on the state Web site, to be implemented 
in 2001. 



It became very evident during our data collection efforts that high stakes educational 
accountability systems are difficult to accurately evaluate from the information available on 
state Web sites. While we intentionally chose to search for Web-based information as an indication 
of the quantity and quality of information readily available to the public at large, it became 
apparent that there is a wide discrepancy among the level of sophistication and degree of 
thoroughness and accuracy of this information from state to state. Additionally, the fact that 
these policies often change due to legislative action, or rulemaking following legislative action, 
would seem to lend itself to updating more frequently on the Web, yet this did not necessarily 
prove to be the case. Some sites seemed to be updated often, and others seemed to be essentially 
unchanged from one visit to another, even though several weeks may have passed in the interim. 
In some instances, the changes were not always improvements. On occasion we would find an 
important document on-line during one visit, and then check the same Web site a few days later, 
and the document would no longer be available, even by a related link. 

It is possible that states indeed have clear guidelines with indicators and formula for their 
accountability systems, but that they do not make them available to the public through their 
Web sites, whether by conscious decision, or simple omission. For example, some states indicated 
that they did collect school and student data in order to make accountability decisions about 
consequences for schools and districts, but they did not explain their process on the Web. Others 
indicated how they were reporting scores for students with disabilities to the public, but did not 
specify whether or how these scores were included in their accountability systems. 

The complexity of the systems that determine accountability may add to the difficulty of making 
information available to the public. Some states face the challenge of duplication of jurisdiction 
of various departments or divisions with overlapping responsibilities for holding schools or 
systems accountable. Some states have created distinct accountability departments or divisions 
that may act independently from state departments of education. Furthermore, states often have 
overlapping systems of accountability. There may be accountability systems, accreditation 
systems, and report card systems, which may or may not be related to each other, or may be 
driven or directed by distinct departments. 
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One of our biggest challenges in studying state accountability systems was determining whether 
specific indicators were actually included in the systems. Many states reported that they collected 
data on multiple indicators, but in fact, relied solely on statewide testing data in determining the 
consequences for districts or schools. When other data were used, they were most often reported 
for comparison purposes rather than applied to a formula used to calculate a school’s or district’s 
score or index. 

Although several states included policy information about student participation in testing and 
the reporting of those scores to the public, they did not include specific information about the 
inclusion of scores in the accountability formula. States often did not specify whether students 
who were excluded from statewide testing were included in accountability systems in some 
other manner, such as by entering zeros for their testing data, or using scores from an alternate 
assessment aligned with the statewide testing, or whether they were entirely omitted from the 
system. One notable exception to this was Kentucky where scores from alternate portfolios are 
included in the academic indices. This allows every student to contribute the same weight to the 
academic component of the accountability index, regardless of whether they participated in the 
regular components of the assessment system or in the alternate portfolio assessment. 

Our research also highlighted the difficulty in distinguishing between various types of 
consequences, particularly between “assistance” and “sanctions.” Though many states classified 
their interventions as being part of an assistance program, there was often a fine line between an 
intervention that was assistance and one that actually crossed the threshold and became a sanction. 
There was little consistency between states on this determination. What was considered assistance 
in one state might be classified to be a sanction in another. Some states used assistance-like 
interventions and sanction-like interventions in reverse order. For example, Nevada mentioned 
punitive results for school failure, including effects on funding and school administration that 
might precede interventions of assistance; assistance would be rendered only after three years 
of failure. 

Despite our original intent to gather information on the extent to which students with disabilities 
were included in school and district accountability systems, we were not very successful in 
doing so. To a great extent, our failure was our desire to obtain publicly available information, 
which was in itself not clear at all. What we did find raises concerns about the extent to which 
states are, or even intend to, include students with disabilities in their accountability calculations. 
We were alarmed to find a state formula that specifically reduced the impact of students with 
disabilities on accountability scores. While the notion of some type of phase-in of requirements 
might seem appropriate for student accountability (i.e., give students with disabilities extra 
time to meet graduation test requirements), it does not seem appropriate for system accountability, 
where the approach, instead, might be to increase the impact of these students on accountability 
scores so that the students get the opportunities that have not been provided to them in the past. 
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Title I requires that all students be included in accountability measures of adequate yearly 
progress. These measures are to be in place by 2001 . Our analysis, which did not seek out these 
adequate yearly progress measures, indicated that existing state accountability systems are far 
from the Title I ideal. It appears that education has a significant way to go before it is truly 
accountable for the learning of all students. 
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Types of Accountability Indicators 
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Other Indicators 


Free/Reduced Lunch 


Enrollment change, 
Accreditation 


Promotion Rate, Safety 


School safety 




Preschool exposure 




Promotion Rate, Free/Reduced 
Lunch 








Postsecondary Education 
Attainment 


Follow up with Graduates 






Participation Rates 


School Safety 




Lowest Quartile, Accreditation, 
Free/Reduced Lunch 
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Other Indicators 
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Lunch, Postsecondary 
Education Attainment 


Poverty Rates 


Student-to-Counselor Ratio, 
Gain/ Growth in Scores, 
Participation Rates, Safety 


Diploma Options, Vocational 
Credentials earned, Post- 
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Other Indicators 
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Tennessee 


Texas 


Vermont 


Virginia 


Washington 


West Virginia 



& ■g 



T3 T3 



24 



NCEO 



= Indicators collected but not used to determine consequences according to the state's Web site 
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Accountability Formulas 



These states indicated on their State Department of Education Web sites the following formulas to used to determine 
which schools or districts were to receive consequences based upon the indicators. 

Louisiana 

A School Performance Score will be calculated for each school based upon its performance on 4 indicators. These 
scores can range from 0 to beyond 100. The 10- Year Goal is set at a score of 100. The 20- Year Goal is set at a score 
of 150. Each indicator will be given the following weights as follows: 



K-6 Schools 



Indicator 


Weight 


LEAP 21 Tests 


60% 


The Iowa Tests 


30% 


Student Attendance 


10% 



7-12 Schools 



Indicator 


Weight 


LEAP 21 Tests 


60% 


The Iowa Tests 


30% 


Student Attendance 


5% 


Drop Out Rate 


5% 



Data for these three/four indicators will be converted using various formulas (that were not provided) to form a 
single School Performance Score. During the summer of 1999, the state will calculate the initial baseline School 
Performance Scores Grades K-8. In the summer of 2001, the School Performance Scores will be calculated for 
Grades 9-12. 

Subsequent School Performance Scores will be the average of two year’s worth of data. For example, in 2001, 
60% of the School Performance Score for a K-6 school will be the average of their spring 2000 and spring 2001 
LEAP 2 1 test scores. 

The School Performance Score is used in the calculation of a school's Growth Target. Schools will receive 
consequences based on whether they meet or exceed their Growth Target. The current formula for the Growth 
Target , to be used for the first 10 years of implementation, is: 

[PropRE x (100-SPS) tN] + [Prop SE x (100-SPS) -r 2N], or 5 points , whichever is greater 

PropRE: proportion of regular education students (including gifted/talented, speech impaired only, & students 

with 504 plans); 

SPS: School's Performance Score; 

N: number of remaining accountability cycles in 10-year period; 

PropSE : proportion of special education students eligible to participate in LEAP 21 and The Iowa Tests 



Mississippi 

The Accreditation System in Mississippi uses a two-phase process to assign performance levels to schools and 
districts. In the first phase schools and districts are compared to 37 Level 3 Performance Standards. Schools and 
districts that meet or exceed 90% of these standards are compared to 38 Level 5 Performance Standards. 

Level 3 Performance Standards 

The first 36 Level 3 Standards are based on test scores derived from Functional Literacy Exam (FLE) subtests; the 
Iowa Tests of Basic Skills/Tests of Achievement and Proficiency (ITBS/ TAP) Survey Battery subtests for grades 
4-9; the Algebra I, U.S. History, and Biology I subject area tests; and the ITBS/TAP Performance Assessments for 
grades 4-9. The following formulas are used to determine the criteria against which school and district scores are 
measured: 
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AMV = State MS - _ SD (for FLE, ITBS/TAP, Algebra I, U.S. History, and Biology I tests) 

AMV = State MS - SD (for the ITBS/TAP Performance Assessments) 

AMV: Annual minimum value that districts test scores must meet or exceed 

State MS: State Mean Score for that test 

SD: Standard Deviation of the student level scores 

[NOTE: If the calculated AMV falls between the absolute minimum value and the maximum value as set by the 
State, this value is used for the year. If the calculated AMV falls short of the absolute minimum, the AMV is set at 
the absolute minimum value. If the calculated AMV exceeds the maximum value, the AMV is set at the maximum 
value.] 

The 37 th Level 3 Standard is the percentage of students with scores below the 25th percentile on 7 of the 10 tests 
(ITBS/TAP Survey Battery Total for grades 4-9; FLE Composite; Algebra I, U.S. History, and Biology I subject 
area tests). For the Level 3 Performance Standards this value is set at 30% for 7 of the 10 tests. 

Level 5 Performance Standards 

The first 36 Level 5 Standards are again based on test scores derived from FLE subtests; the ITBS/TAP Survey 
Battery subtests for grades 4-9; the Algebra I, U.S. History, and Biology I subject area tests; and the ITBS/TAP 
Performance Assessments for grades 4-9. The school’s and district's actual scores on these standards are compared 
to a criterion value set as the mean of all Level 3 Districts. 

The 37 th Level 5 Standard is the percentage of students with scores below the 25th percentile on 7 of the 10 tests 
(ITBS/TAP Survey Battery Total, grades 4-9; FLE Composite; and Algebra I, U.S. History, and Biology I subject 
area tests). For the Level 5 Performance Standards this value is set at 25% for 7 of the 10 tests. 

The 38 th Level 5 Standard is the graduation rate for students in the district. For Level 5 Performance Standards this 
value is set at 75%. 

In addition, schools and districts must meet all of the 61 Process Standards that focus on District Leadership, 
Instructional Focus, Effective Instruction, Professional Development, and School Climate. 



North Carolina 

Schools and districts receive consequences dependent upon their performance and growth/gain composite scores. 

A Performance Composite Score is calculated for each school by determining the total number of scores at or 

above Level III (i.e., at or above grade level) in each subject divided by the total number of valid scores. This 

performance composite score is reported as a percentage. 

• For schools that have any grade 3, 4, 5, 6, 7, 8, or 10 students, the performance composite is the total number 
of students at or above Level III “in reading and mathematics including the North Carolina High School 
Comprehensive Test (NCHSCT), writing, and any course with an EOC test that may be offered at the school.” 

• For schools that offer only high school courses, the performance composite is the percent of students at or 
above Level HI in Algebra I & II, Biology, Chemistry, ELPS, English I & II, Geometry, Physical Science, 
Physics, U.S. History, and the NCHSCT. 

A Growth/Gain Composite Score is calculated for each school by summing the following: 

• End-of-Grade growth (EOG) in reading and math for grades 3-8 and growth in reading and math from 8 th grade 
to 10 th grade (see formula below); 
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• Writing Index gain over baselines from 4 th and 7 th grades using three years of data; 

• End-of-Course (EOC) Indexes gain determined by comparing current performance to an average of the previous 
two years performance; 

• The gain in percentage of current graduates who completed a College Prep/College Tech Prep course compared 
to the baseline average of 1998 and 1999 percentages; and 

• Gains in the percentage of students meeting the North Carolina Competency Standards over the baseline of 
those who scored at Level III on the grade 8 EOG tests in reading and math. 

The total composite score must be greater than zero to meet the expected and exemplary growth/gain standards. 

This formula is used to calculate the school’s two expected growth scores — one for reading and one for math: 

Expected growth = b Q + (b, x ITP) + (b 2 x IRM) 

b Q = North Carolina average rate of growth in the respective grade and subject 
bj = value used to estimate “true proficiency” * 
b 2 = value used to estimate “regression to the mean”* 

ITP =Index for “true proficiency” which is equal to (Local Reading Scale Scores + Local Math Scale Scores) - 
(NC Reading Scale Scores + NC Math Scale Scores) 

IRM(reading) = Index for Regression to the mean which is equal to (Local Reading Scale Scores - NC avg Reading 
Scale Scores) 

IRM(math) = Index for Regression to the mean which is equal to (Local Math Scale Scores - NC avg Math Scale 
Scores) 

• NC avg Scale Scores (1994-95 S Y) are used as the b 0 value. 



Grade 


Reading 


Mathematic 


s 


3 


143.4 


141.2 


4 


147.6 


147.9 


5 


152.4 


154.4 


6 


154.5 


160.2 


7 


158.1 


166.0 


8 


160.1 


170.3 


* Values used to Estimate 




Value 


Reading 


Mathematics 




b, 


0.22 


0.26 


b* 


-0.60 


-0.58 



New York 

The accountability measures used examine test performance for each of the following subjects as indicated: 

• English Language Arts [for grades 4 and 8] = (B+D)/(A+C) 

A = number of all students tested on the State English/Language Arts Exam 
B = number of students at or above Level 2 on the State English/Language Arts Exam 

C = number of ELL students using alternative methods who’s English proficiency is below the participation level 
D = number of ELL students using alternative methods who are making satisfactory progress 

• Mathematics [for grades 4 and 8] = B/A 

A = number of all students tested on the State Math Exam 
B = number of students at or above Level 2 on the State Math Exam 
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• Demonstrated Competency [for grade 11]= (B+C)/A 
A = Grade 1 1 enrollment in June 
B = Regents Level (higher) 

C = RCT level (lower) 

The criterion value for each of these measures is 90%. 



The performance accountability system includes: performance scores, performance standards, designations, the 
report card, continuous improvement process, technical assistance, intervention, performance incentives and rewards, 
and fiscal accountability. District report cards will be distributed to provide information based on performance on 
specific indicators and designate school districts as: 

• Effective = 26 or more indicators 

• Continuous Improvement = 14-25 indicators 

• Academic Watch = 9-13 indicators 

• Academic Emergency = 8 or less indicators 

The first 15 indicators are if 75% of 4 th , 6 th , and 9 th grade students pass the Proficiency tests for the five subjects. 
Indicators 16-20 are if 85% of students are passing the 9 th grade Proficiency test by the 10 th grade. Indicators 21-25 
are if 60% of the 12 th grade students pass the 12 th grade Proficiency tests. The 26 th and 27 th indicators are if the 
school maintains a 93% minimum attendance rate and 90% graduation rate [revised from a max 3% dropout rate] 
respectively. 



Ohio 
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